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* notices * 

JPO and NCIPZ axe not responsible for any 
damages caused by the use of this translation. 

1 .This document has been translated by computer. So the translation may not reflect the original precisely. 
2.**** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



CLAIMS 



[Claim(s)] 

[Claim 1] The 1st camera made to generate at least one image The camera of at least one addition which changed at least one property 
or a parameter with said 1 st camera, Equipment made to generate the synthetic image containing the image processor which is 
connected to said 1st camera and the camera of said at least one addition, and changes said at least one image from said 1st camera 
into other images with which one or more [ of a property or a parameter ] differ. 

[Claim 2] said — others — the claim from which an image is changed by at least one of one or more properties in the camera of said 
addition, or the parameters - equipment given in the 1st term. 

[Claim 3] Equipment given in the 1st term of a patent claim in which said property and parameter include the spatial location of said 
camera. 

[Claim 4] Equipment given in the 1st term of a patent claim in which said property or parameter contains the spatial resolution of said 
camera. 

[Claim 5] Equipment given in the 1st term of a patent claim in which said property or parameter includes the spectral characteristics of 
said camera. 

[Claim 6] Equipment given in the 1 st term of a patent claim in which said property or parameter includes the spatial arrangement of 
the system of coordinates of said camera to carry out. 

[Claim 7] In the approach of compounding an image The step which supplies the 1st resolution image recorded from the 1st resolution 
camera, It is the step which supplies the 2nd resolution image recorded from the 2nd resolution camera. In that case said 1 st resolution 
Step made larger than said 2nd resolution Step which calculates an image flow using two or more images Approach containing the 
step which carries out warping of said 1 st resolution image to the view of said 2nd resolution camera using said image flow which 
generates a synthetic image. 

[Claim 8] Furthermore, step which calculates the parallax evaluation for said low resolution Approach given in the 7th term of a patent 
claim which projects said parallax evaluation and contains a step in order to use, in case a synthetic image is calculated. 
[Claim 9] The step which calculates said parallax evaluation is . An approach given in the 7th term containing the step which 
reinforces said parallax count by performing one or more increment routines chosen from the group who consists of applying the 
technique disassemble an image into a tile, take correlation of the depth, perform the Kachi difference technique exactly, and check 
correspondence, and avoid a null field of a patent claim. 

[Claim 10] In the medium which has memorized two or more commands and in which computer reading is possible It is said processor 
when said two or more commands are performed by the processor. The step which supplies the 1 st resolution image recorded from the 
1st resolution camera, It is the step which supplies the 2nd resolution image recorded from the 2nd resolution camera. In that case said 
1 st resolution Larger step than said 2nd resolution Two or more images are used. An image flow Count step, An approach given in the 
7th term of a patent claim which was made to perform the step which carries out warping of said 1st resolution image to the view of 
said 2nd resolution camera using said image flow which generates composition image. 



[Translation done.] 
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DETAILED DESCRIPTION 

[Detailed Description of the Invention] 
[0001] 

[The opinion of priority] 

This application is the U.S. temporary application for which it applied on August 28, 1998. The profits based on 35USCSectionl 19 of 

No. 60/3 68 are charged, and those contents are used for this specification as a whole. [ 098 and 368 ] 

[0002] 

This application is U.S. application No. for which it applied to coincidence, (patent attorney reference number SAR 13422) The theme 

relevant to an indication is indicated. The contents are used for this specification as a whole. 

[0003] 

This invention relates to the approach and equipment which this invention makes generate the synthetic image of the high resolution 
from two or more cameras with which one or more properties differ from a parameter in more detail about an image processor. 
[0004] 

[Description of the Prior Art] 

In the application of amusement and others, the high resolution stereo image of a scene can be obtained and it is useful so that an 
observer can observe the scene of three dimensions. In order to obtain such a high resolution image, in common with a Prior art, use of 
two or more high resolution equipments or cameras which shifted the mutual location and have been arranged is used. The 1 st high 
resolution camera catches other high resolution images photoed from a different view from this 1st high resolution camera, the image 
compounded, or an image sequence, and generates the stereo image of a scene. 
[0005] 

However, cost starts very much while creation of the stereo image using two or more high resolution cameras is difficult. Hie number 
of the high resolution cameras used in order to record a scene occupies most manufacturing costs of a stereo image. Moreover, a high 
resolution camera is large and it is hard to treat it. Therefore, the ease that a scene is stored in a film can serve as troublesomeness. 
Furthermore, since a photography site may be unable to hold magnitude of such a high resolution camera, there is a limitation in the 
photography site which can create a stereo image. 
[0006] 

Therefore, the approach and equipment which can create one or more synthetic images are desired from two or more cameras with 

which a property differs from a parameter. 

[0007] 

[Means for Solving the Problem] 

The disadvantageous profit looked at by the conventional technique changes into other different images from the 1 st image the image 
recorded with one camera, and is conquered by this invention that consists of the approach and equipment which used the image 
caught with one set or the camera beyond it with which that property and parameter differ from the 1st camera. Some examples 
include the spatial location of a camera, spatial resolution, spectral characteristics, and spatial arrangement. One example of this 
invention is equipment which consists of a low resolution camera made to generate the image of a low resolution connected to the high 
resolution camera and image processor which generate the image of high resolution. An image processor performs various image 
flows and parallax evaluation count, and leaps a high resolution image to the view of a low resolution camera. 
[0008] 

Invention includes the approach materialized as a combination of software and hardware as a software routine. The approach by this 
invention supplies the image of high resolution to image data, supplies the image of a low resolution to image data, processes an 
image, and consists of a step which leaps a high resolution image to the view of a low resolution camera, and subsequently forms a 
synthetic image. Thus, a original high resolution image and a synthetic image are used to form the stereo image of high resolution by 
use of only a single high resolution camera. 
[0009] 

[Embodiment of the Invention] 

Drawing 1 shows the high resolution composition image generator 1 00 by this invention. The input sequence 1 1 2 is supplied to a 
computer 102. The input sequence 1 12 can consist of a frame of the pair at the one time, and a series of frame pairs [ in / spent many 
hours or / a series of frames ]. A computer 102 consists of the central-process unit (CPU) 104, an exchange circuit 106, and memory 
108. The high resolution composition image generating routine 1 10 is located in this memory 108. The high resolution composition 
image generating routine 110 can be read by turns through a network from the source of others like a floppy (trademark) disk, CD, and 
the remote memory source. Furthermore, the computer is connected to input/output accessory 1 1 8. If actuation is explained briefly, the 
input video sequence 1 1 2 will be supplied to a computer 1 02, and a computer will output the synthetic high resolution image 1 14 after 
actuation of the high resolution composition image generating routine 1 10. 
[0010] 

The example of the conversion relevant to the space position of a sensor is the parallax method of recovery described below. The 
example of the conversion relevant to the spatial resolution of a sensor is described by Burt and Adelson in "Laplacian Pyramid as a 
compact Image code", and an image is changed into the resolution of another side from one resolution there in the processing which 
calculates an image pyramid. The example of the conversion relevant to the spectral characteristics of a sensor is mapping to RGB 
(red, green, blue) from HSL (a color, SACHURESHON, brightness), and this is Andrew Glassner. It is as given in "Graphics Gems" 
and Academic Press which were edited. The example of the conversion related to the spatial arrangement to others' spatial 
arrangement of the image recorded from one sensor is indicated by S.Baker and Proceedings of the 6 thlnternational Conference on 
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Computer Vision by S.KNayar, p.35-42, Bombay, India, January, and "A Theory of Catadioptric Image Formation" in 1998. moreover, 
the example of others of conversion related to the spatial arrangement to others' spatial arrangement of the image recorded from one 
sensor - Journal of Computer Vision, Graphics and Image Processing, Vol 14, and p.365- 372 and 1980 — Sandini and Tagliasco - 
"An Anthropomorphic Retina-Like Structure for Scene Analysis" It is indicated. 
[0011] 

As for the high resolution composition image generating routine 1 10 made into a routine 1 10, a detail is henceforth understood from 
drawing 2 in more detail. Although processing of this invention is performed as a software routine 1 10, some of steps by the approach 
indicated here can be performed in hardware the same with being based on a software controller. This invention can be performed in 
the combination of software and hardware in the software performed on a computer system in the hardware as an integrated circuit 
which specified application, or the hardware in other types. Thus, each step of a routine 1 10 must be interpreted as what is equipped 
with the hardware (module) which specified the application of equivalence, or the hardware used in combination with software again. 
[0012] 

The high resolution composition image generating routine 1 10 receives an input 1 12 from a high resolution camera 206 and the low 
resolution camera 208. A high resolution camera 206 observes a scene 200 from the 1st view 216, and another side and the low 
resolution camera 208 observe a scene 200 from the 2nd view 218. The high resolution camera 206 is equipped with image resolution 
higher than the image resolution of the low resolution camera 208. A high resolution camera 206 can consist of equipment with which 
a large number equipped with the data output format from which a large number differ differ, and if this is those who equipped the 
technique concerned with the usual knowledge, it can also apply the equipment, data format, and/or protocol of what kind of number 
by the place which this invention suggests. In one example, a high resolution camera 206 is a high-definition camera like MSM9801 
camera which can be purchased from for example, IMAX. Similarly, since the low resolution cameras 208 can apply a routine 1 10 to 
various equipments easily as mentioned above if they are those who equipped the technique concerned with the usual knowledge, 
various numbers of equipments can constitute them. In one example, the equipment of a low resolution is the camera equipped with 
resolution lower than the resolution of the equipment of high resolution like the video camera of for example, standard image quality. 
For example, image resolution can be equipped with 2 [ at least 8000x6000 pixels / /] cm, and a low resolution image can be equipped 
with 2 [ 1000x1000 pixels //] cm. 
[0013] 

A routine 1 10 receives input data from a high resolution camera 206, and amends spatial aberration, reinforcement-aberration, and 
chromatic aberration in step 202. Chromatic aberration is produced according to the aberration of a lens. The amendment is required in 
order to improve the precision of the continuing step which is performed in a routine 1 10. The approach for calculating the parameter- 
function which has described the aberration function of a lens is common knowledge, for example, H.S.Sawhney and R.Kumar — True 
Multi-Image Alignment and its Application to Mosaicing and Lens Distortion, Computer Vision and Pattern Recogni-tion Conference 
proceedings, and p.450- a parameter is recovered in step 202 using the proofreading procedure indicated by 456 and 1997. The 
contents are used for this specification as a whole. 
[0014] 

Amendment of a color and reinforcement is also performed in step 202. Since the image data from the low resolution camera 208 unite 
with the data from a high resolution camera 206, and unite with any differences which answer the color and reinforcement of a scene 
or produce the artificial structure of the image in the image 1 14 which is required for these amendments since it unites with any 
differences in the equipment by BIGUNETTINGU of a lens, and was moreover compounded as a result, they are required. 
Amendment is performed by proofreading beforehand equipment (namely, a high resolution camera 206 and the low resolution 
camera 208) so that mapping of the color from one equipment to the equipment of a degree and reinforcement may be known. The 
color and on-the-strength amendment information which were measured from each equipment are memorized by the look-up table, or 
are memorized as a parameter-function. 
[0015] 

The input data from the low resolution camera 208 is amended about spatial aberration, reinforcement-aberration, and chromatic 
aberration in step 204 again. The processing which amends the low resolution aberration in step 204 follows up the same processing as 
the amendment performed in step 202. 
[0016] 

While a filter is succeedingly carried out in step 210, the subsample of the high resolution data amended from step 202 is carried out. 
The purpose of step 210 lowers the resolution of a high resolution image, and is in making the resolution of a low resolution image 
match. Since the description which appears in this high resolution image does not appear in a low resolution image, and since an error 
arises in depth recovery (step 306 shown in following drawing 3 ), step 210 is required. When the description exists in one image and 
does not exist in another side, it is essentially tended for especially these errors to be produced, in order that the depth recovery 306 
may try to judge correspondence between a high resolution image and a low resolution image, and, as for this correspondence 
processing, to generate an error. 
[0017] 

Step 210 is first performed by count of the difference of the spatial resolution between a high resolution camera 206 and the low 
resolution camera 208. This is performed as a proofreading step of **** and the scale of relative pixel / cm2 between two cameras is 
calculated. For example, this relative scale is given by the square root of the ratio of the die length in the predetermined configuration 
observed with these two cameras, or surface ratio. A convolution kernel is calculated, and this lessens the high frequency component 
in a high resolution image, and it is made for the remaining frequency component to match those components in a low resolution 
imager from the difference of spatial resolution. This can be performed using the standard sampling theory (for example, The 
Laplacian Pyramid as a Compact Image Code by P.J.Burt and E.H. Adelson which are henceforth used for this specification as a 
whole, IEEE Transactions on Communication, Vol.31 , p.532-540, 1983 reference). 
[0018] 

For example, supposing the high resolution image and the low resolution image are horizontally different in the vertical direction and 
resolution only with a spatial multiplier 2, a suitable filter kernel is [1, 4, 6, 4, 1]. It is /16. Subsequently to the vertical direction, a 
filter kernel is applied horizontally first. The subsample of the high resolution image is carried out by the multiplier 2, and its spatial 
sampling of the image data taken out from the high resolution imager corresponds with it of a low resolution imager. 
[0019] 

In step 210, the filter of the high resolution image data is once carried out, if a subsample is carried out, parallax will be calculated in 
step 212 in each frame time, and the relation between the view 216 in the group of high resolution data and the group of low resolution 
data and a view 218 will be defined. In a detail, parallax count of step 212 calculates more migration of the image pixel between the 
images picturized from the view 216 which is the difference of the view of a scene 200, and the view 218. 
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[0020] 

Since it depends for this parallax information on the relation between two input images equipped with the view (respectively 216 and 
218) when the scenes recorded when the same differ, it is first calculated from the space resolution of a low resolution image. This can 
be performed by re-sampling a high resolution input image using suitable filtering and subsampling processing, as mentioned above in 
relation to step 210. 
[0021] 

Count of step 212 is carried out using the algorithm controlled to some extent based on the assumption performed about the usefulness 
and precision of proofreading information. In the case which is not controlled extremely, a 2-dimensional flow vector is calculated to 
each which is a pixel in an image, and the adjustment is achieved. When epipolar structure is stable and it is known correctly, count 
becomes a single value to each image point. 
[0022] 

When many situations, especially the magnitude of parallax are large, it is effective, if it has set to step 12 and calculates about the 
partial parameter-side of shoes. Parallax count is the count by which the image flow was controlled as a matter of fact. The one 
approach of parallax count is learned as "field plus parallax." This field plus parallax expression can be used to make small the 
magnitude/the amount of pixels which should be evaluated. For example, when a scene 200 is a scene of many cities mostly equipped 
with the plane facet, parallax can be calculated in step 2 1 2 as a combination of the two-dimensional layer equipped with the 
component which separated from the field of an addition of structure. The procedure which performs the field plus parallax approach 
Direct Recovery of Shape From Multiple Views: A Parallax, such as United States patent application No.08/493,632 for which it 
applied on June 22, 1995, and R.Kumar Based Approach 12thICPR, 1974, Harpreet Sawhney, 3D Geometry From Planar Par-allax, 
CVPR94, June 1994, A.Shashua, and N.Navab Relative Affine Structure, Theory and Application to 3D Constr-uction From 2D 
Views, It is indicated by IEEE Conference on Computer Vision and PatternReco-gnition and June 1994 at the detail, and the contents 
are used for this specification as a whole. 
[0023] 

Other algorithms can use in order to perform parallax analysis instead of the field plus parallax approach. ** and thin repeat evaluation 
processing are usually used for these algorithms using the multi-resolution image pyramid expression. These algorithms start 
evaluation of image migration in small resolution, and, subsequently make these evaluations a precision through warping and residual 
migration evaluation in resolution level more detailed than sequential which are repeated. The advantage used as the key in these 
approaches is being able to perform very efficient count which can give sub pixel precision in migration evaluation, even if big 
migration exists. Many publications have described the fundamental technique used in parallax count of step 212. U.S. Pat. No. 
5,259,040 by which the detail of such a technique was registered on November 2, 1993, J. Hierarchical Model-Based Motion 
Estimation by R.Bergen etc., 2 nd EuropeanConference on Computer Vision, Direct Multi-Resolution Estimation of Ego-Motion and 
Structure From Motion p. - 237- it is based on 252, 1992, and KJ.Hanna - IEEE Work-shop on Visual Motion, It Okamoto(es). p. - 
1 56- 162, 1 991 , KJ.Hanna, and Neil E. - Combining Stereo and Motion Analysis for Dire-ct Estimation of Scene Structure, 
International Conference on ComputerVision, Direct Recovery of Shape From Multi-pie Views: A Parallax Based Approach ICPR p. - 
357- it is based on 356, 1993, R.Kumar, etc. - p. - 685- 688 and 1994 - and S. - it is based on Ayer and H.S.Sawhney Layered 
Representati-on of Motion Video Using Robust Maximum-Likelihood Estimation of Mixture Models and MDL Encoding 
International Conference on Computer Vision, p. - 777- it is indicated by 784 and 1995 and the contents are used for this specification 
as a whole. [0024] 

Although it is satisfied with calculating parallax simply using the field plus parallax approach mentioned above of step 212, many 
techniques which can be used in order to make fundamental two-frame stereo parallax count of step 212 into that which is more tough 
and reliable exist. In order to improve the precision of step 212, those techniques are independent or are performed in combination. 
Those techniques contain the routine which is drawn on the block diagram of drawing 3 , and calculates the increment routine 302, the 
Sharp-ized routine 304, and the residual parallax 306, the occlusion detection routine 308, and the motion analysis routine 310. 
Although an improvement of parallax count argues as those with effectiveness, the same increment processing can apply to image 
flow count, and, as for those techniques, is raising the precision of image flow evaluation. 
[0025] 

The increment routine 302 is more tough and makes that reliable fundamental two-frame stereo parallax count. One approach 
disassembles an image into a tile, and in each tile, a display with a parameter is a superior field and it is parallax. Especially a superior 
field is a front field. A display with the even parameter to each tile is controlled by overall rotation and migration (this is solvable by 
being known from proofreading of **** of a stereo setup, or using a direct method). Furthermore, single epipolar control is applied to 
all the parallax vectors of all even tiles. 
[0026] 

A field without the occlusion and the texture from which the other increment routines 302 make parallax count produce an error is 
processed. In order to process a field without occlusion and a texture, it performs from the spatial frequency of that the depth 
coincidence over two frames changes the magnitude of an aperture, and ** to detailed **. In order that the magnitude of many 
apertures may test the coordination of depth evaluation, and the mutually related quality, it is used in predetermined resolution level. It 
is thought that depth evaluation is reliable only when producing the acceptable correlation level in which the magnitude of at least two 
apertures had consistent depth evaluation. Otherwise, the depth of the level is not updated. When the aperture set as the object of 
consideration does not have sufficient texture, if the depth evaluation which is disregarded and has the coordination from the 
magnitude of a big aperture is possible for depth evaluation, it is desirable. The field according to which the field where the depth is 
not defined was filled using the data from the synthetic frame of **♦*, and was made and classified, using pretreatment, or temporary 
prediction using low resolution data, i.e., low resolution data, was sampled, and it was classified in the synthetic image 1 14 is filled. 
The processing using many apertures for improving parallax count is United States patent ****** for which it applied to coincidence. 
It is indicated by the number (patent attorney reference number No.SAR13422), and the contents are used for this specification as a 
whole. 
[0027] 

In order that approach when [ it ] using the increment routine 302 may optimize depth evaluation, the Kachi difference (JND) model is 
used exactly. For example, the scale of a typical image like a difference on the strength is used in order to quantify the error of a depth 
expression. However, those scales are compensated by the observer by JND measurement which tries to measure the error which can 
be checked most by looking. The approach of using the JND approach is explained below at a detail. 
[0028] 

The additional increment routine 302 offers the algorithm which calculates image position correspondence. First, all potential 
correspondences in an image position are most defined by the rotation and migration of a predetermined camera in the range of distant, 
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and, subsequently correspondence is continuously checked in the point location corresponding to the near range from sequential. The 
coordination during the correspondence recovered between adjoining range gives the scale of precision over this correspondence 
[0029] 

The other increment routines 302 do not give the null field in the perimeter of the compounded image. Since it appears in a location 
which a high resolution image is leaped and is different, the image boundary of the compounded image is not in agreement with a 
original composition image. Such a field leaves a null potentially. This problem is solved using three approaches. The 1 st approach is 
displaying only the main aperture of the high resolution image which is an inverse image, therefore the field which poses a problem is 
not displayed. The 2nd approach is filled up with a bordering field using the data from the frame compounded beforehand. The 3rd 
approach is filtering the data from low resolution equipment, carrying out a rise sampling, and inserting the data in an image boundary. 

[0030] 

The additional increment routine 302 offers the algorithm which gives overall 3D and local (multi) field control. The approach 
expressing a flow is between the frame pairs as a smooth parameter residual flow (software control is performed ranging over a tile) 
tile-ized especially. Furthermore, a tile can express in a small number of parameter-layer per tile. When there is overall 3D control 
ranging over two frames (stereo), a tile is expressed as an even layer to which two or more fields can exist in a patch. 
[0031] 

The approach of others for improving the quality of the parallax count in step 212 is using the Sharp-ized routine 304. For example, in 
the discontinuity range or near other transition, there is a typical field of the parallax evaluated in-between because of spatial exchange 
of the finite used in computation 212. The detection which such transition carried out clearly, and continuing "Sharp-ization" of the 
parallax field make those errors small. As expansion of this fundamental processing, the information from the part in early stages of an 
image sequence (and potentially anaphase) is used in order to improve composition of the high resolution image 1 14. For example, the 
detail of the image in an occlusion field is observable from high resolution equipment in the frame which precedes or continues. This 
information needs to be used in order to calculate the information on the motion on a frame from a frame like count of parallax. 
However, count of this addition is performed when not the continuation base under processing of a whole sequence but amendment 
with error is required. 
[0032] 

Furthermore, parallax count of step 212 improves by calculating residual parallax (depth) using an approach equivalent to calculating 
the approach or the residual parallax 306 described below. When it can use under the condition that the stereo sequence which is 
moving is the high resolution quiescence image formatted 15 to 65 times, for example, one approach calculates the coordination of the 
depth about time amount, in order to control the depth / disparity count further. While control of stability was effective, as it described 
the outline about two images caught by coincidence, it can use for two-frame count of the depth. An optical flow is calculated to many 
stereo frames by inter-frame [ which corresponds about time amount ]. An optical flow functions as a prediction means of the depth in 
a new frame. Within a new frame, depth count is performed by making it a pair, receiving control by the soft control which comes 
from the predicted depth evaluation. This is performed to the front and back in time. Therefore, although evaluation is effective in a 
point temporarily, it can be filled up with the field which is not so at other times about both times. 
[0033] 

Other approaches of calculating the residual parallax 306 are using optical flow control for coincidence along with control of the 
stability over depth / disparity count about many stereo pairs. Especially, when most scenes 200 are stability, in depth count, it is 
expressed as a parameter similarly to the time of coincidence having temporary stability control and 2 inter-frame having control of 
stability, and accuracy. When there is an element which is moving separately in the scene 200, the optical flow control about time 
amount can be used as control soft as a part of multi-time amount instant depth count. 
[0034] 

Other approaches of calculating the residual parallax 306 are maintaining coordination while they control the depth as coordination 
about time amount and improve the adjustment over a temporary sequence. For example, if the depth is recovered about a point 
temporarily, the depth in the following frame time can once be predicted by gap of the depth by the rotation and migration of a camera 
which were recovered by the old frame and inter-frame [ new ]. This approach can be developed so that the location of the appearance 
identified appropriately again or an occlusion boundary may be extended and parallax or flow count may be improved. 
[0035] 

Other approaches which calculate the residual parallax 306 are giving the control which continues in order to aim at solution direct for 
a temporary smooth stereo rather than to solve the instant depth, therefore to make a result smooth. This can be developed using 
epipolar one and flow control which were combined. For example, it is possible to use the frame currently compounded beforehand. 
The condition of having been given to the newly compounded frame is smooth in time about having coordination in instant parallax 
count, and the frame generated before, and things. The latter condition is given predicting the flow base based on a former frame, and 
by forming the difference from the prediction part and error term eye. Similarly, when the sequence has already occurred, the frame 
(namely, leaped high resolution image) of the parallax base can be compared with the frame inserted temporarily [ the flow base ]. 
This comparison can be used, also in order to detect a problem field and to refine parallax count. Even if this approach does not 
assume stability, it can be used, and even if it does not make it relate to structure / parallax control, it can be used. In the case of the 
latter, count of the flow base can be performed about a residual motion, after the part of immobilization is amended. This is extensible 
so that two-dimensional control may be applied ranging over a frame along with the overall fixed motion control covering all the 
pixels in one frame. 
[0036] 

Furthermore, the approach of calculating the residual parallax 306 which avoids the potential problem by the instability of the 
synthetic stereo sequence in the three-dimensional structure constituted using the synthetic image 1 14 is restricting the amount of the 
depth change in inter-frame. In order to abolish this problem, it is important to avoid temporary fluctuation in the parallax structure 
extracted using temporary smoothing. The easy form of this smoothing is acquired by only restricting the variation introduced when 
updating former evaluation. In order to perform this systematically, INTAFUREMU motion analysis as well as activation of the 
parallax count in a frame is required. 
[0037] 

The occlusion detection 308 is assistance when the field of the visual field which should be compounded cannot be seen from the 
location of a high resolution camera. When such, it is required to use another source to the image information in the field. Before 
performing this, it must detect that such a situation has occurred. This is obtained by calculating the result obtained when 
correspondence of an image is calculated in both directions. Occlusion is the evaluated gap which is obtained from calculating 
correspondence on either side and calculating coincidence of correspondence on either side in the field which is not a problem. In the 
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field of occlusion, these are not usually in agreement This makes an approach detect an occlusion field. An occlusion condition can be 
predicted from the structure of parallax field itself again. When this is stable to extent related with time amount, the field which 
seemingly has occlusion can be marked into a former frame. Subsequently, a bidirectional technique can use in order to check a 
condition. 
[0038] 

The motion analysis 310 improves parallax count of step 212 again. The motion analysis 3 10 includes the frame frame motion analysis 
in the caught sequence. This information can be used in order to solve occlusion, since it was visible when the fields which are not 
visible in a point temporarily were others (or it comes to be visible). Furthermore, the problem of temporary stability is avoided by 
searching for the three-dimensions-structure which is consistent ranging over some frames of a sequence. 
[0039] 

Frame frame motion analysis consists of usually decomposing observed image change into change (namely, motion of a camera) of a 
view, three-dimensions-structure, and the component by motion of a body. In order to perform this reconstruction, collection of the 
technique for performing evaluation of each component exists. These techniques consist of a layer extract which makes the contents 
the field of the depth of a direct camera motion, motion parallax evaluation, a coincidence motion and the body that parallax-evaluates, 
and moves and is carried out, or many. The component used as the key of these techniques is a "field plus parallax" expression. In this 
approach, parallax structure is expressed as a residual parallax map per [ which makes the contents change of the caused motion move 
and according to plus and a local plane structure which the field (or other parametric representation sides) was caused ] pixel. In the 
count approach, a camera motion moves to the case given by the stereo base line of immobilization, and the parallax evaluation 
technique mentioned above is a special case fundamentally [ analytical skill ]. 
[0040] 

In order to improve processing efficiently, parallax count (or flow count) can be performed in the resolution of a low resolution image. 
Subsequently, parallax information forms the map which is projected and corresponds in higher resolution. Subsequently continuing 
image warping and/or configuration processing are performed using the projected parallax information. 
[0041] 

It is used in order to generate the high resolution composition image [ in / in this / step 214 ] 114, once the parallax field is calculated 
in step 212. A configuration and the warping step 214 are shown in drawing 2 , and the detail is shown in drawing 4 R> 4. Notionally, 
this processing consists of two steps, parallax insertion and image warping. These two steps are usually combined with one actuation 
as shown at step 214 in operation. As opposed to each pixel in the image which should be compounded in the case of which count of 
step 214 The access [ to the movement vector which pinpoints the location in the high resolution source image from a high resolution 
camera 206 ], and ** side of some of (steps 502) and pinpointed locations Access to a certain pixel, And it consists of count (step 506) 
of the insertion value over the compounded pixel which constitutes the synthetic (step 504) image 1 14 based on those pixels. Step 214 
must be performed in the image resolution of a full target. Moreover, for reservation of the required image quality in the synthetic 
image 1 14, the insertion step 506 uses congruence primary or a congruence Miyoshi interpolation function at least, and must be 
performed. 
[0042] 

Still more efficient warping and configuration algorithm move, and can use for the information on parallax and others (step 508). For 
example, it is used in order that the location of the depth discontinuity from depth recovery may prevent the spatial insertion in 
warping over such discontinuity. Such insertion generates blurring in such a field. Furthermore, an occlusion field can be filled up with 
the information from a former frame or the former frame which follows using warping of the flow base. The above-mentioned 
technique in the argument on field plus parallax can be used in order to make step 508 successful. 
[0043] 

Moreover, temporary scintillation of the compounded image can be eliminated using the flow information which gives temporary 
smoothing (step 510). Inter-frame [ in the sequence by which this flow information is compounded ] can exist in all between an 
inverse image and a synthetic image. This scintillation is avoidable with a peak of the appearance descriptor of the pyramid base to the 
compounded field which is equipped with the field where a original high resolution frame corresponds again. These are graduated 
about time amount and a "texture flicker" is removed. 
[0044] 

A configuration and the warping step 2 14 are not from near [ small ] the pixel again, and it performs using the data collected over the 
image patch. For example, an image is divided into the field to which a large number were divided, therefore a re-sampling is 
performed about the part covered by the field in a target image (step 5 1 2). 
[0045] 

Recovery of depth cannot produce the perfect and precise depth evaluation in each pixel. This produces a difference between the 
values produced from required reinforcement or the chromaticity value, and the original high resolution image. Subsequently to the 
following, a warping module can choose one or more [ of the option described as an amendment technique ] separately or together 
(step 514). 

- Leave a result as it is (step 516). 

- Insert the data by which the rise sample was carried out from the low resolution image (step 5 1 8). 

- Use the data compounded beforehand (step 520). 

- The combination of the selection with the Kachi difference (JND) technique suitable exactly (step 522) by which an operator corrects 
a problem manually can be used. A JND technique can be performed to the sequence compounded by comparing the difference 
between the data from the low resolution camera which makes the JND map in which the low resolution configuration of an image, 
and the image quality and the parallax count scale which were compounded are shown, it applied for various JND techniques on 
United States patent application No.09 / March 28, 1 997 [ 055, 076, and ] for which it applied on April 3, 1 989 - said - it applied on 
No.08/829,540, and March 28, 1997 - said - it is indicated by No. 08/82 8, 161, United States patent No.5,738,430, and United States 
patent No. 5,694,491 . ** is used for this specification as a whole among these. Furthermore, after JND is leaped by the flow field 
calculated from the parallax count in step 212, it is carried out between the compounded high resolution image data and the high 
resolution image compounded beforehand. 

[0046] 

If a high resolution composition image is formed in a low resolution view, in order that a original high resolution image and the 

compounded image may form a high resolution stereo image, it will once be used. 

[0047] 

Although the example by the place which this invention suggests was shown and it was explained to the detail, correcting many others 
based on the technical range of these suggestion and this invention will be being able to carry out easily if it is those who have the 

http://www4Jpdl.ncipi.gojp/cgi-biii/tran_web_cgi_ejje 3/22/2006 



JP,2002-524937,A [DETAILED DESCRIPTION] 



Page 6 of 6 



usual knowledge on the technique concerned. 
[Brief Description of the Drawings] 

He can understand easily the place which this invention suggests by taking into consideration the following detailed explanation 
performed with reference to an attached drawing. 
[Drawing 1] 

It is the block diagram of the image formation equipment relevant to the image analytical method and equipment by this invention. 
[Drawing 2] 

It is the block diagram of the image formation equipment used for one example of this invention, and image analytical method. 
[Drawing 3] 

It is the flow chart of the parallax count approach. 
[Drawing 4] 

It is the flow chart of the image composition approach. 

In order to make an understanding easy, if possible to the same member common to a drawing, the same reference number will be 
used for it. 
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